Learning animal social behavior from trajectory features

Automatically classifying behavior of humans and animals from video is one of the most interesting and challenging fields of computer vision, [3, 1, 6]. Most of the successful human behavior recognition works use as features for classification information extracted from a direct representation of the scene (visual features), as opposed to indirect representations such as silhouettes, body parts, pose or object positions, which can be very sensitive to viewpoint variation and occlusions in real-world videos [10]. In contrast, indirect representation of the scene is widely used in the case of animals [1, 6, 4]. Animal enclosures allow for a more controlled filming, which reduces viewpoint variations and occlusions, facilitating the extraction of indirect representations of objects in the scene. Moreover, animal bodies are less expressive than humans, therefore causing direct visual features to work worse on animals [2]. One of the most widely used features for animal behavior recognition is the position of animals in time (result of either manual annotations or an object detection+tracking algorithm) [1, 6, 4, 2]. From the positions, usually several trajectory features are computed, such as distance between animals, their direction of movement, velocity, acceleration, etc. These trajectory features are used, together with the behavior labels, to train a supervised classifier that learns the discriminative features across behaviors. In scenarios where videos are previously segmented 1, such as in KTH [11] or Hollywood2 [8] human datasets, a classic supervised classifier such as SVM [14] or AdaBoost [5] is often used. In more realistic scenarios, however, the task is to fully segment a continuous video into behavior intervals (behavior category, starting frame, ending frame). Most works on animal behavior recognition fall into this category [1, 6, 4, 2], while recent effort has also been made in the human action recognition community to move in this direction, e.g. Virat dataset [9]. In this scenario, more intricate classifiers are needed. In [6] authors use a two layer SVMHMM, while in [2] authors extend Auto-context [13] to video. These classifiers are able to detect behavior classes and at the same time learn behavior transitions, successfully segmenting long, continuous videos into smooth behavior intervals. In this work, we describe a novel extension of Structured SVM and benchmark it against the Auto-context method to measure its robustness and versatility. The rest of the paper is as follows: Section 2 briefly presents the two methods to be compared, Section 3 presents results on two different datasets (mice and flies) and discusses the results.